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I ha  Effects  of  EeedbacK  and  Eifidictabllity  on  Judgment 

Betty  S.  Goldsberry 

\  Abstract 

\ 

Previous  research  has  found  that  when  subjects  are 
given  cognitive  feedback ,  they  reach  higher  levels  of 
achievement  than  when  they  are  given  outcome  feedback.  It 
was  hypothesized  that  this  finding  was  due  in  part  to  the 
predictability  of  the  task  environment  since  outcome 
feedback  is  at  a  distinct  disadvantage  as  a  sole  means 
of  conveying  such  information.  A  study  was  conducted  to 
compare  response  and  outcome  feedback  under  three 
predictability  conditions.  The  design  included  a  control 
group  receiving  no  feedback  at  all,  two  response  groups 
differing  in  precision  of  feedback  information ,  and  two 
outcome  feedback  groups  differing  on  a  quantity  dimension. 

Task  predictability  conditions  averaged  across  five  learning^ 
blocks  were  high  (x  *■  .94),  moderate  (x  ■  .87)  and  low  ^ 
low  (x  ■  .71).^The  study  also  attempted  to  clarify  the 
definition  of  feedback  and  to  equate  the  availability  of  task 
information  in  the  various  feedback  conditions  that  were 
compared. 

Contrary  to  expectations,  the  utility  of  outcome 
feedback  was  inferior  to  that  of  response  feedback  under  all— ^ 


/ 


'three  predictability  conditions  tested.  In  fact,  an 
interaction  revealed  that  the  effect  of  increased 
predictability  raised  rather  than  lowered  the  disparity 
between  outcome  and  response  feedback  performance.  The 
results  also  revealed  that  a  control  group  receiving  no 
feedback  at  all  performed  as  well  as  or  tetter  than 
those  with  feedback  when  the  availability  of  task 
information  was  equated.  Moreover,  eliminating  the 
memory  requirement  inherent  in  the  use  of  outcome  feedback 
only  worsened  performance.  Similarly,  adding  precision  to 
the  response  feedback  condition  beyond  the  level  of  mere 
directional  error  information  did  not  improve  performance. 

ft 

The  principal  conclusions  to  be  drawn  from  these  \ 
findings  are:  (a)  increasing  predictability  improves 
judgment  performance  but  does  not  enhance  the  effectiveness 
of  outcome  feedback,  (b)  providing  outcome  feedback  is 
actually  detrimental  to  performance  when  the  subject  is 
adequately  instructed  regarding  the  underlying  task 
structure,  and  (c)  increasing  the  precision  of  response 
feedback  beyond  mere  direction  of  error  is  of  no  apparent 
value  in  multiple-cue  judgment  tasks. 
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Introduction 

It  is  now  well  established  that  human  judgment 
capabilities  are  limited,  and  that  decisions  based  largely 
on  intuition  can  be  seriously  biased  or  flawed  (Nisbett  6 
Ross,  1980).  Further,  people  are  poor  at  recognizing  the 
deficiencies  in  their  own  performance,  and  as  a  result, 
tend  to  be  over  confident  (Einhorn  &  Hogarth,  1981;  Slovic 
et  al.,  1977).  Attempts  to  improve  both  performance  and 
awareness  through  the  use  of  systematic  training  procedures 
have  met  with  only  limited  success  (Slovic,  1982). 

Naturally,  a  central  feature  in  most  such  "debiasing" 
paradigms  is  a  provision  of  feedback.  Since  knowledge  of 
results  (or  feedback)  has  long  been  considered  a 
sufficient— if  not  necessary— condition  for  learning 
(Holding,  1959),  the  question  of  why  it  is  not  more 
generally  effective  in  moderating  judgment  and  decision 
behavior  has  been  of  interest  for  some  time  (Fischhoff, 
1975)  ,  Among  the  conclusions  that  have  emerged  are  (a)  the 
recognition  that  in  many  common  judgment  situations, 
feedback  is  insufficient,  irrelevant,  or  even  misleading 
(Einhorn,  1980)  and  (b)  the  observation  that  only  certain 
types  of  feedback  are  useful  when  the  relationships  to  be 
learned  are  probabilistic  rather  than  deterministic  (as  is 
usually  the  case  in  judgment  and  decision  tasks) .  More 
specifically,  it  has  been  suggested  that  the  mere  knowledge 
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of  the  outcomes  produced  by  a  judgment  or  decision  process 
(i.e.  their  accuracy,  level  of  success,  payoff,  etc.)  is 
relatively  useless  or  even  counterproductive,  whereas 
information  regarding  the  pricess  itself  (i.e.  the  task 
structure,  the  ideal  response  strategy,  or  both)  can  produce 
improvement  (Deane  et  al.,  1972;  Hammond  &  Summers,  1965; 
Hoffman  et  al.,  1981;  Summers  &  Hammond,  1966). 

Although  considerable  evidence  has  been  gathered  in 
support  of  the  above  generalisations,  the  issue  of  how  to 
structure  feedback  for  purposes  of  improving  and/or 
sustaining  judgment  performance  is  far  from  resolved.  For 
one  thing,  the  outcome-process  distinction  is  but  one  of 
many  that  have  been  applied  to  the  feedback  concept: 
feedback  can  be  manipulated  in  a  host  of  ways,  all  of  which 
could  have  implications  for  judgment  performance.  For 
another,  it  has  recently  been  pointed  out  that  task 
characteristics  in  addition  to  the  feedback 
itself— independently  and  in  conjunction  with  feedback— can 
influence  the  efficacy  of  any  particular  kind  of  feedback. 
Adelman  (1981),  for  example,  has  shown  that  congruence, 

or  the  degree  of  correspondence  between  implied  and  actual 
properties  of  the  task  environment  determine  the  relative 
effectiveness  of  outcome  and  process  (or  cognitive) 
feedback:  outcome  feedback  is  not  only  useful,  but  as 
effective  as  cognitve  feedback  when  incorporated  into  a 
highly  congruent  task.  At  the  risk  of  oversimplification. 
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what  this  means  is  that  a  decision  maker  (DM)  can  benefit 
from  outcome  feedback  if  he/she  has  a  good  conception  of  the 
processes  by  which  outcomes  are  produced;  otherwise,  such 
information  only  leads  to  confusion.  Cognitive  feedback,  on 
the  other  hand,  is  useful  for  acquiring  that  understanding, 
but  redundant  once  the  basic  structure  is  learned.  Thus  in 
a  congruent  task  (where  DM  is  already  familiar  with  the 
basic  processes) ,  outcome  feedback  serves  as  well  as 
cognitive  feedback  in  maintaining  performance. 

The  present  study  was  designed  to  explore  further  the 
relative  efficacy  of  outcome  feedback  in  judgment  as  a 
function  of  task  conditions.  In  this  case,  however,  every 
effort  was  made  at  the  outset  to  insure  that  DM  was  aware  of 
the  process  or  rule  by  which  outcomes  (criterion  values)  were 
related  to  the  predictive  information  (cue  values) .  Such 
conditions  would  be  present  in  any  real-world  judgment  task 
where  cue-criterion  relations  were  known.  The  task  property 
of  interest  here  was  task  predictability,  or  the  extent  to 
which  the  "process*  relating  cues  to  outcomes  was  reliable. 
The  main  iscue  was  whether  feedback  type  would  interact  with 
this  task  feature  in  a  manner  similar  to  that  found  by 
Adelman  (1981)  for  congruence.  That  is,  does  the 
effectiveness  of  outcome  (versus  cognitive)  feedback 
increase  with  task  predictability  as  it  does  when  the  task 
becomes  more  congruent?  In  a  sense,  both  manipulations 
could  be  viewed  as  ways  of  making  the  judgment  task  easier. 


Although  several  investigations  have  speculated  upon  a 
predictability-feedback  interaction  (Adelman,  1981;  Payne, 
1982),  none  has  yet  demonstrated  it. 

Thus  the  principal  hypothesis  addressed  in  the  present 
study  was  that  outcome  feedback  should  become  more  effective 
relative  to  cognitive  feedback  in  shaping  and  sustaining 
judgment  performance  as  task  predictability  increases. 

Since  all  subjects  were  familiar  with  the  task  structure 
(hence  "congruence"  was  fixed  at  a  high  level) ,  the 
cognitive  feedback  dealt  primarily  with  the  appropriate 
response  strategy.  That  is,  DM  presumably  knew  what  the 
cue-criterion  relationships  were,  and  thus  his/her  only 
concern  was  how  to  produce  responses  in  a  manner  consistent 
with  this  structure.  The  feedback  indicated  the  degree  of 
correspondence  between  the  response  strategy  evidenced  in 
DM's  judgment  behavior  and  the  optimal  strategy  (therefore 
the  cognitive  feedback  conditions  are  referred  to  as 
response  feedback  in  the  remainder  of  this  report) . 

The  task,  analytic  approach,  and  interpretations 
involved  in  this  study,  like  those  of  its  predecessors,  all 
draw  heavily  upon  the  so-called  Brunswik  Lens  Model  of 
judgment  (Brunswik,  1952,  1955).  Therefore,  a  brief  review 
of  this  model  is  in  order. 

In  essence,  the  "lens  model"  (illustrated  in  Figure  1) 
separates  characteristics  of  the  environment  from 
characteristics  of  the  human  judge.  The  left  portion  of 
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Figure  1  represents  the  environment  and  illustrates  the 
"true”  relationship  between  the  predictive  cues  and 
criteria,  whereas  the  right  portion  represents  the  judge  and 
illustrates  cue-criterion  judgment  relationships.  The  left 
portion,  therefore,  permits  a  normative  analysis  of  judgment 
while  the  right  portion  permits  a  descriptive  analysis. 

Tucker  (1964)  suggested  that  the  relationship 
between  the  judgments  and  the  criteria  could  be 
partitioned  into  several  statistically  independent 
components  reflecting:  (a)  the  judge's  acquired  knowledge 
of  task  properties,  (b)  his/her  cognitive  control  in 
applying  uhat  knowledge,  (c)  the  degree  of  predictability  in 
the  task  environment,  and  (d)  the  nonlinearity  in  the 
judgments.  The  equation  reads  as  follows: 

R.  -  G  RsRe+  C  Jl  -  R//l  -  R/  U> 

where 

-  the  relationship  (correlation)  between  the 
judgments  and  the  criteria: 

G  ■  the  correlation  between  the  linear  predictions 
of  the  judgments  and  the  linear  predictions  of  the 
criteria; 

Rs  •  the  correlation  between  the  judgments  and  the 
linear  predictions  of  the  judgments; 

Rfi  *  the  correlation  between  the  criteria  and  the 
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Pigure  1:  Prunsvik's  Lens  Model 
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linear  predictions  of  the  criteria ,  a  measure  which  places 
an  upper  limit  on  achievement; 

C  ■  the  correlation  between  the  variance  in  the 
task  system  and  the  variance  in  the  response  system,  a 
measure  which  is  an  indication  of  nonlinearity  in  the 
judge's  response  strategy. 

Hammond  and  Summers  (1972)  used  the  lens  model  to 

isolate  two  of  these  components — the  acquisition  of 

knowledge  (G)  and  the  application  of  knowledge  {Rg). 

They  defined  acquisition  as  the  extent  to  which  the 

judge's  cognitive  system  is  isomorphic  with  (in  the  same 

• 

form  as)  the  task  environment.  They  defined  application, 
or  cognitive  control,  as  the  extent  to  which  acquired 
knowledge  is  utilized  consistently  in  making  judgments. 

The  isolation  of  these  two  factors  made  possible  an 
assessment  of  how  feedback  type  and  other  task 
characteristics  affect  performance  through  use  of  multiple 
regression  analysis. 

In  this  application,  regression  is  used  to  model  the 
way  in  which  information  about  the  cues  is  rsed  or  should  be 
used  to  produce  a  judgment.  It  accomplishes  this  by 
generating,  from  an  intercorrelation  matrix  of  cue 
dimensions,  a  linear  regression  equation  that  indicates  how 
best  to  weight  each  cue  dimension.  If  the  cue  values  are 
regressed  on  the  "correct"  judgments  (criteria) ,  the  linear 
model  illustrates  an  optimal  weighting  strategy  (a  normative 


model  of  judgment);  and  if  they  are  regressed  on  the 
"observed"  judgments,  the  model  illustrates  a  response 
weighting  strategy  (a  descriptive  model  of  judgment  or  the 
judge's  "policy”).  If,  for  example,  the  weight  of  a  cue 
dimension  was  1.0,  it  would  indicate  that  the  judge  has 
relied  completely  on  that  dimension  in  making  his  judgments, 
whereas  a  zero  weight  would  indicate  that  he  has  ignored  the 
dimension  completely. 

The  present  research,  then,  used  the  lens  model  and 
multiple  regression  analysis  to  create  a  theoretical 

■  •  i 

framework  for  the  investigation  of  the  effects  of  feedback 
type  and  task  characteristics  on  human  judgment.  Since  the 

I 

requisite  knowledge  (i.e.  proper  cue-weighting j strategy  or  G) 

was  furnished  directly  through  instructions,  ai^y  effects  of 

I 

the  manipulations  were  expected  to  appear  in  terms  of  the 

I  ■  ■ 

application  of  knowledge  (i.e.  cognitive  contrdl  or  R_) 

i  ■  8 

and  the  overall  achievement  index  (i.e.  R_). 

s 


Method 


Subjects -flnd-Egsisa 

Seventy  undergraduate  psychology  students 
participated  in  the  research  project  as  judges  (subjects) 
in  exchange  for  extra  course  credit  or  $12.00  in  cash. 
Subjects  were  randomly  assigned  to  five  treatment  groups 
defined  on  the  basis  of  feedback  type.  They  made  judgments 
on  three  different  hypothetical  jobs,  each  representing  a 
different  level  of  predictability.  The  distinction  between 
these  five  feedback  groups  and  three  jobs  is  explained  in 
greater  detail  in  the  next  section. 

To  control  for  possible  order  effects,  presentation 
of  the  three  jobs  was  counterbalanced;  to  minimize 
fatigue  effects,  each  job  was  presented  at  a  different 
session.  Each  session  was  divided  into  a  warm-up  period 
and  five  practice  blocks.  The  experimental  design, 
therefore,  was  a  mixed  model  5  (feedback  type)  by  3  (task 
predictability)  by  5  (practice  block)  factorial  with  14 
subjects  per  group. 

Task 

The  judgment  task  was  chosen  on  the  basis  of  its 
common  usage  in  human  judgment  research  and  the  likelihood 
that  it  would  be  meaningful  for  a  wide  variety  of 
potential  subjects.  It  consisted  of  evaluating  the  overall 
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suitability  of  hypothetical  job  applicants  based  upon  their 
ratings  on  three  skill  dimensions  (i.e.  the  cue  values). 
Using  the  weighting  strategy  shown  in  Table  1,  subject  were 
required  to  integrate  the  three  cue  values  for  each  of  320 
applicants  into  a  single  suitability  rating  on  a  scale  of  1 
(least  suitable)  to  9  (most  suitable) .  Since  this  weighting 

TABLE  1 

Optimal  Weighting  Strategy 

Skill  Rating 
1  2  3 


Regression  Weight  .50  .30  .20 


strategy  was,  by  definition,  the  normatively  optimal  model 
of  the  task  environment  (left  portion  of  the  lens  model), 
it  was  used  to  generate  the  criteria  and  the  criterion 
predictions  characterizing  the  task.  Random  error  was  then 
added  to  the  criteria  to  produce  the  three  levels  of 
predictability  (one  for  each  job):  (a)  high,  in  which 
JL  ■  .94,  (b)  moderate,  in  which  x.  ■  .87,  and  (c)  low,  in 
which  l  *  .71.  Therefore,  three  different  sets  of  criteria 
were  generated  and  only  one  set  of  predictions.  In  the 
low  predictability  condition,  50  percent  of  the  variance 
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was  due  to  random  error;  in  the  moderate  and  high 
predictability  conditions,  the  corresponding  error 
variances  were  24  and  12  percent,  respectively.  These 
three  sets  of  criteria  were  used  as  the  basis  for  generating 
feedback  in  all  four  of  the  feedback  groups. 

Each  applicant  profile  (illustrated  in  Figure  2) 
contained  three  skill  ratings  and  a  set  of  irrelevant 
biographical  information.  It  was  presented  via  a  Visual 
200  terminal  controlled  by  an  Advanced  Systems/9000 
computer.  The  skill  ratings  were  generated  orthogonally 
from  a  normal  distribution  of  numbers  ranging  from  1  to  9 
with  a  mean  of  5  and  a  variance  of  2.  Biographical  data 
were  randomly  selected,  from  the  Houston,  Texas  telephone 
directory. 

The  five  types  of  feedback  were:  (a)  no  feedback 
(control),  (b)  non-historical  outcome  feedback, 

(c)  historical  outcome  feedback,  (d)  comparative  response 
feedback,  and  (e)  exact  response  feedback.  In  the  control 
condition  (a) ,  the  subject  was  forced  to  rely  entirely  upon 
the  strategic  information  provided  by  the  instructions  (see 
Table  1):  there  was  no  opportunity  for  judgment-to-judgraent 
calibration  as  in  the  four  feedback  conditions.  The  two 
outcome  feedback  conditions  (b  and  c)  afforded  the  subject 
knowledge  of  the  "correct"  response  as  generated  by  the 
environmental  model  (including  the  random  error  component). 
They  differed  in  that  non-historical  feedback  (b)  indicated 


Name:  Mary  Frances  Smith 

522  Pontiac  Avenue 
Houston,  Texas  77024 

Telephone  No.  567-3443 


Rating  Skill  No.  Rating 

1  7 

2  2 

3  5 


Your  response  is 


Figure  2:  Profile  format 
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the  correct  response  for  the  current  applicant  only,  whereas 
historical  feedback  (c)  displayed  the  results  for  the  past 
20  applicants  as  shown  in  Figure  3.  This  manipulation  was 
designed  to  control  for  the  role  of  memory  in  any  obtained 
outcome-cognitive  feedback  difference.  That  is,  if  the 
typical  inferiority  of  outcome  feedback  is  due  to  the 
subject's  inability  to  remember  how  previous  responses 
turned  out,  as  some  have  suggested,  then  historical  outcome 
feedback  should  ameliorate  the  deficiency. 

Finally,  the  two  cognitive  (response)  feedback 
conditions  (d  and  e)  provided  the  subject  with  information 
on  how  his  cue-weighting  policy  over  the  last  20 
judgments  (right-hand  side  of  the  lens  model)  compared  to 
optimal  (left-hand  side  of  the  lens  model) .  It  was  obtained 
by  regressing  the  actual  judgments  onto  the  cue  values 
(ratings)  and  displaying  the  resulting  beta  weights  either 
numerically  in  comparison  to  the  optimal  ones  (e  above  as 
illustrated  in  Figure  4) ,  or  in  comparative  terms  (d  above 
as  illustrated  in  Figure  5) .  In  the  latter  case,  the 
"tolerance  interval"  for  a  correct  response  ("OK"  feedback) 
was  a  captured  weight  set  to  within  ±.05  units  of  the 
optimal  weight  for  a  particular  cue. 

Procedure 

All  subjects  participated  in  three  sessions,  each 
approximately  60  minutes  in  length  and  scheduled  one  week 
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Figure  4:  Exact  re* 
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Rating  Skill  Opt: 

Lmal  Weighting 

Your  Weighting 

1  | 

.50 

Too  high 

2 

.30 

OK 

3  .20  Too  Low 


Figure  5:  Comparative  response  feedback  display. 
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apart.  At  the  beginning  of  each  session ,  written 
instructions  were  given  describing  the  job,  the  assessment 
procedure,  the  feedback  type,  and  the  optimal  weighting 
strategy.  To  insure  full  understanding  of  this 
information,  instructions  were  augmented  by  a  graphical 
illustration  of  how  each  skill  dimension  correlates  with 
on-the-job  performance.  The  subjects  were  also  taught  how 
to  use  the  Visual  200  terminal  to  enter  their  judgments. 

Subjects  were  told  that  each  session  would  be 
devoted  to  making  suitability  ratings  on  applicants  for 
three  different  jobs.  The  normality  and  dependence 
features  of  the  profiles  were  also  explained  so  the 
subjects  would  not  be  misled  searching  for  nonexistent 

profile  structures.  The  actual  .,0b  titles  and  skill 
dimensions,  however,  were  not  identified  so  that  subjects 
would  not  be  influenced  by  prior  knowledge  or  familiarity 
with  the  jobs. 

All  subjects  were  paced  through  20  warm-up  profiles 
followed  by  300  experimental  ones  with  the  aid  of  a  tape 
recording  of  "beeps"  presented  at  10-second  intervals. 
Between  each  set  of  20  profiles  (a  unit)  there  was  a 
60-second  pause  during  which  the  control  and  non-historical 
groups  rested  and  the  other  three  groups  received  their 
end-of-unit  feedback . 

tl£3SUI£S 
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The  scaled  judgments  served  as  the  basis. for  two  sets 
of  derived  measures:  two  product  measures  (hit-rate  and 
achievement)  and  two  process  measures  (knowledge  and 
cognitive  control) .  The  product  measures  indicated  how 
closely  judgment  performance  approximated  the  defined 
optimal  while  the  process  measures  examined  the  inferred 
cognitive  elements  underlying  that  performance. 

The  model  adopted  for  this  purpose  was  the  standard 
linear  regression  approach  commonly  used  in  human  judgment 
research  and  explained  in  the  Introduction.  The  optimal 
weights  assigned  to  the  various  skill  dimensions  are  shown 
in  Table  1.  Hit-rate  was  simply  the  proportion  of  a 
subject's  judgments  that  matche  the  output  of  the  "true"  or 
ecologically  valid  weighting  model.  Achievement  was  the 
correlation  between  the  subject's  judgments  and  the 
optimal  model's  "judgments."  Knowledge,  or  the  subject's 
.  »r standing  of  the  optimal  weighting  strategy,  was 
indexed  by  the  correlation  between  the  criteria  produced 
by  the  optimal  weighting  strategy  and  judgments  produced 
by  a  model  of  the  subject's  weighting  strategy.  The 
latter,  of  course,  was  derived  from  the  subject’s  actual 
judgments  through  the  use  of  multiple  regression  analysis 
to  "capture"  his  policy.  Control  was  indexed  by  the 
correlation  between  judgments  predicted  on  the  basis  of 
the  subject's  captured  policy  and  ones  he  or  she  actually 
produced. 
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The  mathematical  relationship  between  achievement  and 
the  process  measures  (knowledge  and  control)  was  discussed 
in  the  Introduction.  For  present  purposes,  this 
relationship,  which  is  a  mathematical  statement  of  the  lens 
model  as  set  forth  in  equation  1,  is  simplified  as  follows: 

Ra  =  G  Rg  Rg  (2) 

because  specification  of  an  optimal  weighting  strategy 
makes  the  criterion  as  predictable  as  Rgf  and  the  optimal 
strategy  is  linear  (eliminating  the  need  for  the  right-most 
term  in  equation  1) .  The  simplified  equation  renders  the 
distinction  among  measures  used  in  the  present  research 
apparent.  The  term  represents  achievement  which  can  be 
partitioned  into  knowledge  (G) ,  control  (R  ) ,  and 
predictability  (Re) .  Judgments  are,  therefore,  accurate 
(Rfl)  to  the  extent  that  they  correspond  with  the  a  tual 
suitability  of  the  applicants  as  reflected  by  the 
substantive  properties  of  the  task.  It  follows,  therefore, 
that  a  subject  can  be  accurate  in  his  judgments  to  the 
extent  that  he  has  a  predictable  task  structure  (Rg) ,  he 
understands  the  structure  (G) ,  and  he  is  capable  of  using 
that  understanding  consistently  (R  ) . 

9 

Analysis 


Product  and  process  measures  were  determined  in 
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keeping  with  the  definitions  just  presented.  Since  task 
predictability  is  a  component  of  performance  as  well  as  an 
independent  variable  in  this  research #  absolute  and  relative 
measures  of  performance  were  calculated  and  analyzed.  The 
absolute  performance  measures  were  the  observed  block  scores 
while  the  relative  performance  measures  were  calculated  by 
dividing  each  observed  block  score  by  the  optimal  block 
score  (obtained  by  applying  the  optimal  weighting  strategy 
to  the  cue  values).  Naturally#  the  optimal  block  scor* 
declined  as  predictability  was  reduced.  It  also  varied 
somewhat  across  learning  blocks  since  predictability  was  not 
counterbalanced  over  blocks  in  this  study.  Tables  2  and  3 
show  the  optimal  block  scores  used  to  calculate  the  relative 
achievement  and  hit-rate  measures#  respectively. 

Separate  analysis  of  variance  procedures  were 
performed  on  absolute  and  relative  measures  in  order  to 
assess  the  significance  of  main  effects  and  interactions 
among  the  independent  variables.  Dunnett  tests#  which 
compare  treatment  means  to  a  control  group#  were  also  used 
to  compare  the  effects  of  each  feedback  type  (with  the  no 
feedback  control.  In  addition#  Newman-Keuls  tests  of 
paired  comparisons  were  carried  out#  when  appropriate#  to 
isolate  the  pattern  of  significant  effects.  These  tests 
were  performed  in  a  manner  described  by  Winer  '1962) . 
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Table  2 

Optimal  Achievement  Scores 

Total  Job  Task  Predictability  (x.) 


High 

Moderate 

Low 

Mean 

Blocks 

1 

.83 

.87 

.72 

.81 

2 

.95 

.91 

.80 

.87 

3 

.97 

.87 

.61 

.82 

4 

.98 

.92 

.78 

.90 

5 

.96 

.86 

.64 

.82 

— 

— 

— 

— — 

Mean 

.94 

.87 

.71 

.84 

Table  3 


Optimal  Bit-cate  Scores 
Task  Predictability  Levels  {%  correct) 


High 


Moderate  Low 


Blocks 
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Results  and  Discussion 

For  purposes  of  clarity  in  exposition,  the  principal 
findings  are  organized  around  four  major  questions  addressed 
by  this  research:  (a)  Does  task  predictability  influence 
performance?  (b)  Does  feedback  type  influence  performance? 

(c)  Does  practice  influence  performance?  and  (d)  Does 
feedback  efficacy  differ  as  a  function  of  task 
predictability?  The  data  bearing  on  each  of  these  questions 
will  be  preceded  by  a  brief  discussion  of  the  expected 
results  and  their  relevance  to  human  judgment.  Findings  will 
be  based  on  analyses  of  variance  performed  on  the  absolute 
hit-rate  (Ha) r  relative  hit-rate  (Hr),  absolute 
achievement  (R&j ,  relative  achievement  (Rr) ,  knowledge 
(G) ,  and  control  (Rg)  measures  of  performance  which  were 
described  in  detail  earlier. 

Feedback  types  will  be  represented  by  the  following 
abbreviations:  (a)  T,  for  the  control  group  which  was  given 

only  task  information,  (b)  N,  for  the  non-historical  outcome 
group,  (c)  H,  for  the  historical  outcome  group,  (d)  C,  for 
the  comparative  response  group,  and  (e)  E,  for  the  exact 
response  group. 

1.  Does  task  predictability  influence  performance? 

Task  predictability  can  have  two  different  types  of 
influence  on  performance.  First,  as  an  independent  variable 


and  means  of  manipulating  a  substantive  property  of  the 
judgment  taskf  predictability  can  alter  the  performance  of 
the  optimal  weighting  strategy.  A  reduction  in 
predictability ,  therefore,  should  produce  a  performance 
decrement  even  if  the  subject  makes  accurate  and  consistent 
usage  of  the  available  task  information.  Second, 
predictability  can  have  an  effect  on  the  way  in  which 
subjects  process  the  information  presented  to  them.  This, 
of  course,  is  the  more  interesting  influence  from  a 
psychological  standpoint.  By  contrast,  the  first  influence 
is  important  in  that  it  serves  as  an  indication  of  the 
subject's  sensitivity  to  the  manipulation— in  essence  a 
method  check. 

As  noted  in  the  Method  section,  two  different  kinds 
of  performance  measures  (absolute  and  relative)  were  used  to 
explore  the  effects  of  predictability  on  judgment.  The 
influence  of  predictability  as  a  manipulator  of  the  task 
content  was  investigated  by  analyzing  the  absolute  hit-rate 
and  achievement  measures.  A  significant  effect  for  these 
measures  would  suggest  that  subjects  were  trying  to  use  the 
optimal  weighting  strategy  to  make  their  judgments  or  at 
least  that  they  were  sensitive  to  the  manipulation.  The 
influence  of  task  predictability  as  a  cognitive  component 
of  performance  was  investigated  by  analyzing  the  relative 
performance  measures  since  they  reflect  performance  after 
manipulation  effects  have  been  removed.  Significant 
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effects  for  both  absolute  and  relative  measures  were 
anticipated. 

In  general,  the  predictability  manipulation  had  the 
desired  effect  on  performance,  for  as  predictability 
increased,  so  did  performance.  Although  this  finding  comes 
as  no  surprise,  it  is  nonetheless  important  because  it 
suggests  that  subjects  were  sensitive  to  this  task  property 
and  were  trying  to  maximize  their  performance.  As 
illustrated  in  Figure  6,  both  absolute  product  measures  of 
performance  yielded  significant  effects  for  task 
predictability,  £  (2,  130)  -  672.15,  a  <  .01  for 
achievement,  and  £  (2,  130)  ■  377.70,  a  <  >01  for 
hit-rate. 

Predictability  was  also  found  to  have  a  significant 
effect  on  relative  performance.  The  two  relative  indices 
did  not  agree,  however,  on  the  nature  of  this  effect. 

Figure  7  indicates  that  relative  achievement  increased  with 
an  increase  in  predictability,  £  (2,  130)  ■  8.01, 
a  <  *01,  and; that  relative  hit-rate  decreased  with  an 
increase  in  predictability,  £  (2,  130)  -  45.63,  a  <  *01. 

The  ambiguity  suggested  by  the  relative  measures 
could  be  an  artifact  of  the  way  accuracy  was  defined  for 
hit-rate:  only  an  exact  match  between  the  subject's 
judgment  and  the  value  produced  by  the  optimal  weighting 
model  was  considered  a  "hit."  Consider  the  optimal 
hit-rate  values  shown  in  Table  3.  Under  high 
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predictability,  the  model  was  100  percent  correct  by 
definition.  Since  even  a  v  11-calibrated  subject  would 
have  trouble  approximating  this  level,  given  that  a 
judgment  had  to  be  perfect  to  be  considered  correct,  one 
would  expect  relative  hit-rate  to  be  rather  low.  Under  low 
predictability,  on  the  other  hand,  the  model  was  correct 
only  about  40  percent  of  the  time,  a  considerably  easier 
standard  against  which  to  express  the  subject's 
performance.  Even  random  responding  would  have  yielded  a 
relative  hit-rate  of  nearly  25  percent  under  thi3  condition 
(compared  to  about  11  percent  under  high  predictability) . 
Had  a  mere  lenient  criterion  been  set  for  the  definition  of 
a  "hit,"  such  as  a  one-unit  confidence  interval  around  the 
model's  judgments,  relative  hit-rate  might  have  produced  a 
trend  more  similar  to  that  of  relative  achievement. 

The  process  measures,  knowledge  and  control,  provide 
insight  into  the  manner  by  which  cognitive  aspects  of 
judgment  affect  performance.  As  illustrated  in  Table  4, 
they  reveal  that  when  predictability  is  increased  from 
the  low  to  the  moderate  level,  both  the  understanding  and 
the  application  of  task  structure  information  increases; 
but  when  it  is  increased  further  to  the  high  predictability 
level,  no  additional  improvement  in  cognitive  processing 
occurs.  Although  small  in  absolute  terms,  the  difference 
between  low  and  moderate  predictability  yielded  a 
significant  effect  of  task  predictability  for  both 
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TABLE  4 

Mean  Process  Measures  under 
Three  Predictability  Conditions 

Measures  (£.) 


Knowledge 

Control 

Predictability  Level 

Low 

.88 

.86 

Moderate 

.89 

.87 

High 

.89 

.87 

— 

— 

Mean 

.89 

.87 
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knowledge,  £  (2,  130}  »  4.92,  p  <  .01,  and  control, 

E.  (2,  130)  -  3.70,  p  <  .03.  Hence,  an  increase  in  task 
predictability  can  significantly  improve  these  two  aspects 
of  cognition;  but  if  they  are  already  at  a  limit  dictated 
perhaps  by  mental  capability  or  capacity,  no  further 
improvement  will  occur. 

Task  predictability,  therefore,  had  a  significant,  though 
very  small,  effect  on  judgmental  knowledge  and  control.  As 
predictability  increased,  so  did  these  aspects  of  judgment, 
although  perhaps  limited  by  a  "ceiling"  associated  with  the 
particular  properties  of  the  task. 

2.  Does  feedback  type  affect  performance? 

The  present  study  was  designed  to  control  for  the 
typical  confounding  of  task  knowledge  and  feedback  type.  In 
previous  studies,  only  subjects  receiving  cognitive  feedback 
had  access  to  specific  (and  important)  knowledge  of  the 
formal  task  structure.  By  providing  such  information  to  all 
feedback  groups  via  instructions,  the  present  design 
permitted  a  fair  comparison  of  response  and  outcome  feedback 
conditions.  The  comparison  of  each  feedback  type  with  a 
control  group  receiving  no  feedback  (task  information  only), 
therefore,  isolated  its  utility  in  judgment.  In  light  of 
recent  findings,  all  feedback  types  tested  were  expected  to 
provide  some  benefit,  at  least  under  some  conditions,  with 
response  feedback  generally  being  more  beneficial  than 


■'  ■)  \ 


■  >;•'  1 


30 


outcome  feedback . 

The  results  revealed  that  the  type  of  feedback 
presented  had  a  significant  overall  effect  on  performance 
for  both  absolute  indices,  £  (4,  65)  =  7.82,  p  <  .01  for 
achievement  and  E.  (4,  65)  =  4.21,  p  <  .01  for  hit-rate. 
However,  contrary  to  expectations,  Table  5  illustrates  that 
the  two  response  feedback  groups  performed  at  about  the  same 
level  as  the  control  group,  but  all  three  were  better  than 
the  two  outcome  feedback  groups.  This  relationship  was 
supported  by  the  results  of  Dunnett  tests  which  revealed 
that  for  absolute  measures,  performance  under  exact  and 
comparative  conditions  (response  feedback)  was  not 
significantly  different  from  the  control  (p  >  .05) y  but 
historical  and  non-historical  conditions  (outcome 
feedback)  were  significantly  inferior  to  the  control 
(p  <  .05) .  The  results  of  Dunnett  tests  also  revealed  that 
for  relative  indices,  only  historical  group  performance  was 
significantly  inferior  to  the  control  (p  <  .05) .  These 
findings  generally  challenge  the  utility  of  all  four 
types  of  feedback  tested  when  subjects  are  informed  of  the 
proper  weighting  strategy  prior  to  their  judgments.  It 
does,  however,  corroborate  the  evidence  that  outcome 
feedback  is  detrimental  to  judgment  performance. 

It  is  apparent  from  the  relative  performance  measures 
that  when  the  effect  of  predictability  as  an  aspect  of  task 
content  is  removed,  the  relationships  between  feedback 
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Table  5 

Mean  Product  Measures  under 
Five  Feedback  Conditions 

Measures 


Absolute  Relative  (ratios) 


Ha<%)  Ra(D  Hr  Rt 


Feedback  Types 
No  feedback 


Control 

50 

.76 

.77 

.90 

Outcome  feedback 

Non-historical 

43 

.71 

.69 

.84 

Historical 

40 

.67 

.66 

.79 

Response  feedback 

Comparative 

47 

.75 

.69 

.88 

Exact 

50 

.75 

.75 

.88 

— 

— 

— 

— 

Mean 

47 

.73 

.71 

.86 

»  »  . 
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types  is  unchanged.  Significant  feedback  type  effects 
appeared  for  both  relative  achievement ,  £.  (4,  65)  «  7.51, 

B  <  .01),  and  relative  hit-rate  measures,  £.  (4,  65)  ■  2.86, 

B  <  .03. 

A  comparison  of  the  two  types  of  outcome  feedback 
suggests  that  preserving  historical  outcome  data  (cues, 
judgments,  and  criteria  for  the  past  20  profiles)  hurts 
•ather  than  helps  the  decision  maker.  This  finding  was 
supported  by  a  Newman* Keuls  test  that  revealed  a  significant 
difference  in  absolute  achievement  between  the  two  outcome 
feedback  groups  (b  <  .05) .  Relative  achievement  was  also 
lower  when  a  history  was  available,  but  the  difference  was 
not  significant.  It  will  be  recalled  that  this  manipulation 
was  introduced  in  order  to  determine  whether  deficiencies  of 
memory  could  explain  the  previously  reported  inferiority  of 
outcome  feedback.  If  the  decline  was  due  to  failure  of 
memory,  performance  should  have  been  better  with  the 
historical  record.  Since  the  trend  was  in  the  opposite 
direction,  failure  of  memory  would  not  seem  a  reasonable 
explanation. 

The  detrimental  effect  of  historical  outcome  feedback 
can,  however,  be  explained  with  reference  to  the  process 
measures.  Inspection  of  the  knowledge  and  control  measures 
in  Table  6  reveals  that  the  subjects  receiving  a  history  of 
their  judgments  exhibited  less  control  over  their  response 
strategies  than  did  those  without  such  a  history.  A 
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Table  6 

Mean  Process  Measures  under 
Five  Feedback  Conditions 

Measures  (jl) 

Knowledge  Control 


\ 

Feedback  Type 
No  feedback 


Control 

.90 

.92 

Outcome  feedback 

Non-historical 

.87 

.86 

/ 

/ 

Historical 

.87 

o 

CO 

. 

\ 

Response  feedback 

.  \ 
> 

Comparative 

at 

00 

. 

.88 

Exact 

.90 

.89 
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Newman-Keuls  test  supported  this  conclusion ,  the  mean 
difference  of  .06  between  these  two  outcome  feedback 
conditions  being  highly  significant  (ft  <  .01)  for  the 
control  index.  By  contrast ,  the  results  for  the  knowledge 
index  were  identical  whether  or  not  history  was  displayed. 
Thus  the  decline  in  performance  is  clearly  attributable  to 
the  effect  that  historical  information  has  on  the  subject's 
ability  to  apply  his  or  her  own  policy  consistently. 

Turning  to  the  more  constructive  feedback  types ,  it 
appears  that  feedback  with  the  precision  of  regression 
weights  leads  to  no  better  performance  than  comparative 
information  based  on  those  regression  weights.  This  is 

j 

supported  by  the  fact  that  a  Newman-Keuls  test  yielded  no 
significant  difference  (ft  >  .05)  for  t|he  two  response 

I 

groups  on  any  of  the  measures  analyzed.  Regression  weights, 

I 

therefore,  are  probably  simplified  to  some  extent  during 
cognitive  processing.  This  research  does  not  permit 

i 

speculation  on  the  degree  of  simplification  that  takes  place 
but  it  does  suggest  that  regression  weights  are  no  more 
useful  than  comparative  information  derived  from  them,  at 
least  under  the  conditions  studied  here. 

In  summary,  feedback  was  found  not  to  have  the 
expected  positive  effect  on  performance.  Response  feedback 
yielded  performance  similar  to  no  feedback  at  all,  while 
outcome  feedback  yielded  considerably  poorer  performance. 

A  principal  consequence  of  outcome  feedback  was  its 
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detrimental  effect  on  cognitive  control ,  a  detriment  which 
was  exacerbated  by  the  addition  of  historical  information 
(i.e.  prior  outcome  data). 

3.  Does  practice  influence  performance? 

Learning  a  cognitive  skill  such  as  the  one  used  in 
this  research  has  been  conceptualized  as  a  three  stage 
mental  process  (Fitts,  1964).  The  first  stage  involves  an 
initial  encoding  of  the  skill  into  a  form  sufficient  to 
generate  the  desired  behavior  to  some  crude  extent  (i.e. 
rule  learning) .  This  stage  is  characterized  by  rapid 
learning  and  sometimes  verbal  mediation  or  rehearsal  while 
the  task  is  being  attempted.  The  second  or  "associative" 
stage  involves  the  "smoothing  out"  or  perfecting  of  the 
sk ill  performance.  This  stage  is  characterized  by  a  slowing 
down  of  learning  while  gradually  detecting  and  eliminating 
errors  in  the  initial  understanding  of  the  skill. 
Concomitantly,  there  is  a  dropout  of  verbal  rehearsal.  The 
third  or  "autonomous”  stage  involves  even  slower  but 
continued  improvement  in  performance  over  a  long  period  of 
time. 

When  the  subjects  in  this  experiment  were  presented 
with  an  optimal  weighting  strategy  and  given  an  opportunity 
to  practice  using  it  prior  to  the  experimental  trials, 
the  intent  was  to  focus  on  the  latter  of  these 
stages — eliminating  for  the  most  part  the  early 
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rule-learning  stage  which  has  been  the  emphasis  in  most 
probability-learning  research.  Therefore,  results  were 
expected  to  reveal  a  gradual  but.  consistent  improvement  in 
performance  across  blocks,  or  at  the  very  least  sustained 
application  of  the  rules  learned  at  the  outset.  The 
justification  for  this  orientation  was  the  practical 
consideration  that  in  most  real-world  decision  systems,  DM 
would  be  appraised  of  any  known  cue-criterion  relations. 

No  attempt  was  made  to  counterbalance  predictability 
over  blocks  and,  as  a  result,  there  was  a  degree  of 
confounding  between  these  variables,  as  illustrated  in 
Table  2  (maximum  achievement  possible).  However,  average 
predictability  was  approximately  equal  for  the  three  most 
widely  spaced  blocks  (jl  -  .81,  .82,  and  .82  for  blocks  1, 

3,  and  5,  respectively) .  Consequently,  analysis  of 
practice  effects  was  limited  to  these  three  blocks  in  order 
to  control  predictability. 

Looking  first  at  the  achievement  index  (Figure  8) ,  the 
results  show  the  anticipated  gradual  improvement  in  only 
three  of  the  five  feedback  groups:  control,  non-historical 
outcome,  and  comparative  response  conditions.  The  historical 
outcome  and  exact  response  groups  showed  an  increase  on 
block  3  and  a  decrease  on  block  5.  an  effect  which  may  have 
been  due  to  the  increased  mental  load  or  stress  imposed  by 
these  feedback  conditions.  In  any  case,  analysis  revealed  a 
significant  effect  of  practice  on  absolute  achievement, 
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£  (2,  130)  -  5.89,  a  <  .01,  but  mean  performance  collapsed 
across  the  five  groups  did  not  suggest  a  typical  learning 
function  (.69,  .71,  and  .70  respectively  for  the  three 
blocks).  Rather,  the  above-mentioned  distinction  between 
feedback  type  functions  was  supported  by  a  significant 
type  by  block  interaction,  £  (8,  130)  »  2.19,  a  <  .03.  The 
trends  for  hit-rate  (Figure  9)  were  similar  in  form  to  those 
for  achievement,  but  the  feedback  type  by  blocks  interaction 
did  not  reach  significance,  £  (8,130)  ■  1.15,  a  <  .34. 

Turning  to  the  component  process  measures  (knowledge 
and  control),  improvements  occurred  only  in  blocks  1  and  3, 
and  none  of  the  interactions  with  feedback  approached 
significance.  Figure  10  shows  that  knowledge  averaged  across 
groups  increased  from  G  »  .79  in  I: lock  1  to  G  ■  .92  in  block 
3  but  declined  slightly  in  block  5  (G  ■  .90), 

£  (2,  130)  •  507.32,  a  <  .01;  Figure  11  shows  a  less 
dramatic  but  still  significant  trend  for  control, 

F  (2,  130)  ■  4.95,  a  <  .01*  The  sharp  increase  in 
knowledge  between  blocks  1  and  3  suggests  that  perhaps  the  ^ 
rule-learning  stage  was  not  completed  in  the  warm-up  period  1 
as  planned  and  that  subjects  were  still  encoding  task 
structure  information  to  some  extent  during  the  early 
blocks. 

Taken  together,  these  results  suggest  that  practice 
has  a  significant  effect  on  performance  but  the  nature  of 
the  effect  is  specific  to  both  the  feedback  type  and  the 
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measure  used  to  evaluate  it.  In  addition ,  early  practice 
influences  performance  by  improving  the  subject's 
understanding  of  task  structure  (knowledge)  more  so  than  his 
or  her  application  of  that  knowledge  (control) . 

4.  Does  feedback  efficacy  differ  as  a  function  of 
predictability? 

It  was  hypothesized  that  the  difference  between  the 
effects  of  outcome  and  response  feedback  would  diminish  as 
predictability  was  increased.  Response  feedback  was 
expected  to  yield  high  performance  at  all  three  levels  of 
predictability  tested,  while  outcome  feedback  was  expected  to 
do  so  only  when  predictability  was  high.  Therefore,  the 
difference  between  outcome  and  response  feedback  performance 
was  expected  to  decline  as  predictability  increased, 
resulting  in  a  feedback  type  by  predictability  interaction. 

Findings  did  not  support  this  hypothesis  even  though  a 
feedback  type  by  predictability  interaction  was  obtained  in 
the  absolute  hit-rate  measure,  £.  (8,  130)  *  4.40,  p.  <  .01.  ~ 
As  illustrated  on  Figure  12,  the  direction  of  this 
interaction  effect  was  exactly  opposite  that  predicted  (i.e. 
the  differences  among  feedback  conditions  increased  with 
predictability) .  A  similar  trend  occurred  for  the  absolute 
achievement  measure  (Figure  13) ,  but  the  effect  was  not 
significant,  E.  (8,  130)  -  1.57,  r  <  .14. 

The  relative  measures  produced  somewhat  less  consistent 
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findings  than  did  the  absolute  measures.  Relative  hit-rate 
(illustrated  in  Figure  14)  showed  a  sharp,  negative  relation 
between  predictability  and  performance,  whereas  achievement 
(Figure  15)  showed  a  gradual,  positive  one.  Whatever  their 
shape  and  direction,  however,  the  effects  of  feedback 
conditions  on  these  functions  were  consistent  with  the 
absolute  measures.  Performance  was  degraded  by  outcome 
feedback  and  the  expected  interaction  did  not  materialize. 
Only  that  for  relative  hit-rate  was  significant, 

E.  (8,  130)  =  2.42,  (p  <  .02).  Apparently,  the  decline  in 
relative  achievement  for  the  outcome  feedback  conditions 
(Figure  15)  was  not  large  enough  to  cause  a  feedback  by 
predictability  interaction,  E  (8,  l'*'*)  »  .58,  p  <  .79. 

Taken  together,  these  results  suggest  that  unlike  task 
congruence,  task  predictability  does  not  increase  the 
comparability  of  cognitive  (response)  and  outcome  feedback 
effects.  Rather,  it  would  appear  that  outcome  feedback 
becomes  more  detrimental  to  judgment  performance  as 
predictability  increases,  and  is  generally  inferior  to  no 
feedback  at  all,  regardless  of  whether  performance  is 
measured  in  absolute  or  relative  termb. 


Conclusions 


The  present  study  shows  that  feedback  efficacy  is 
influenced  by  task  predictability,  but  not  in  the  way 
suggested  by  Adelman's  hypothesis.  Merely  knowing  the  rule 
by  which  predictive  information  is  related  to  consequent 
outcomes  does  not  insure  that  DM  will  make  good  use  of 
feedback  either  for  applying  that  rule  more  consistently  or 
for  maintaining  a  level  of  performance.  In  fact,  subjects 
did  show  improvement  over  successive  trial  blocks  (largely 
as  a  function  of  improved  consistency) ,  but  not  because  of 
the  contribution  of  feedback:  the  no-feedback  control 
subjects  did  equally  as  well  as  the  best  feedback  subjects 
at  all  levels  of  predictability. 

What  feedback  did  contribute  was  all  negative,  and 
that  negative  contribution  increased  as  the  task  became  more 
predictable.  In  particular,  outcome  feedback  hurt 
performance  and  did  so  more  seriously  as  task  predictability 
increased,  much  as  if  the  "noise"  (or  unpredictable) 
component  were  amplified  by  DM  relative  to  the  "signal" 
component  as  task  conditions  improved.  Preserving  records 
of  past  outcomes  and  responses  only  served  to  worsen  the 
situation  rather  than  "dampen  out”  the  fluctuations.  On  the 
other  hand,  response  (cognitive)  feedback  produced  no 
decrement  regardless  of  task  predictability.  Presumably, 
whatever  cognitive  representation  of  the  proper  weighting 
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strategy  was  established  at  the  outset  was  left  undisturbed 
by  both  metric  and  directional  feedback.  Of  course,  since 
neither  kind  of  feedback  produced  any  differential 
improvement  (relative  to  the  control  condition),  it  was 
clearly  not  necessary  for  either  the  preservation  or 
reinforcment  of  that  cognitive  representation. 

There  remains,  then,  a  discrepancy  between  the 
influence  of  two  task  properties  (congruence  and 
predictability)  on  the  efficacy  of  feedback  in  general  and 
outcome  feedback  in  particular.  Making  DM's  task  clearer  by 
increasing  congruence  apparently  promotes  the  usefulness  of 
outcome  feedback;  doing  so  by  improving  the 
"signal-to-noise"  ratio  (predictability)  only  promotes  the 
harmfulness  of  outcome  feedback.  Since  these 
generalizations  derive  from  separate  studies,  the  next  step 
toward  clarification  would  seem  to  lie  in  the  direction  of 
concurrent  investigation. 
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Defense  Advanced  Research  Projects 
Agency 

1400  Wilson  BlvU. 

Arlington,  VA  22209 


Dr.  M.  D.  Monteraerlo 
Human  Factors  &  Simulation 
Technology,  RTE-6 
NASA  HQS 

Washington,  D.C.  20546 


Ocher  Organizations 


Ms.  Denise  Benel 
Essex  Corporation 
333  N.  Fairfax  Street 
Alexandria,  VA  22314 


Dr.  Andrew  P.  Sage 
School  of  Engineering  and 
Applied  Science 
University  of  Virginia 
Charlottesville,  VA  22901 
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Other  Organizations 

Dr.  Robert  R.  Hackle 

Human  Factors  Research  Division 

Canyon  Research  Group 

S77S  Dawson  Avenue 

Coleta,  CA  93017 

Dr.  Amos  Tversky 
Department  of  Psychology 
Stanford  University 
Stanford,  CA  94305 

Dr.  H.  Mcl.  Parsons 
Essex  Corporation 
333  N.  Fairfax 
Alexandria,  VA  22314 

Dr.  Jesse  Orlansky 
Institute  for  Defense  Analyses 
1801  N.  Beauregard  Street 
Alexandria,  VA  22311 


Other  Organizations 

Dr.  Harry  Snyder 
Dept  of  Industrial  Engeerlng 
Virginia  Polytechnic  Institute 
and  State  University 
Blacksburg,  VA  24061 

Dr.  Stanley  Deutsch 
NAS-National  Research  Council  (COHF) 
2101  Constitution  Avenue,  N.W. 
Washington,  D.C.  20418 

Dr.  Amos  Freedy 
Perceptronics,  Inc. 

6271  Variel  Avenue 
Woodland  Hills,  CA  91364 

Dr.  Robert  Fox 
Department  of  Psychology 
Vanderbilt  University 
Nashville,  TN  37240 


Dr.  J.  0.  Chinnis,  Jr. 

Decision  Science  Consortium,  Inc. 

7700  Leesburg  Pike 
Suite  421 

Falls  Church,  VA  22043 
Dr.  T.  B.  Sheridan 

Department  of  Mechanical  Engineering 
Massachusetts  Institute  of  Technology 
Cambridge,  MA  02139 

Dr.  Paul  E.  Lehner 
PAR  Technology  Corp. 

P.0.  Box  2005 
Reston,  VA  22090 

Dr.  Paul  Slovic 
Decision  Research 
1201  Oak  Street 
Eugene,  OR  97401 


Dr.  Meredith  P.  Crawford 
American  Psychological  Association 
Office  of  Educational  Affairs 
1200  l 7th  Street,  N.W. 

Washington,  D.C.  20036 

Dr.  Deborah  Boehm-Davis 
General  Electric  Company 
Information  &  Data  Systems 
1755  Jefferson  Davis  Highway 
Arlington,  VA  22202 

Dr.  Howard  E.  Clark 
NAS-NRC 

Commission  on  Engrg.  &  Tech.  Systems 
2101  Constitution  Ave. ,  N.W. 
Washington,  D.C.  20418 
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Or.  Charles  Geccys 
Department  of  Psychology 
University  of  Oklahoma 
455  West  Lindsey 
Norman,  OK  73069 

Or.  Kenneth  Hammond 
Institute  of  Behavioral  Science 
University  of  Colorado 
Boulder.  CO  80309 


Dr.  Babur  M.  Pulat 
Department  of  Industrial  Engineering 
North  Carolina  A&T  State  University 
Greensboro,  NC  27411 

Dr.  Lola  Lopes 

Information  Sciences  Division 
Department  of  Psychology’ 

University  of  Wisconsin 
Madison,  WI  53706 


Dr.  James  H.  Howard..  Jr. 
Department  of  Psychology 
Catholic  University 
Washington,  D.  C.  20064 


National  Security  Agency 
ATTN:  N-32,  Marie  Goldberg 
9800  Savage  Road 
Ft.  Meade,  MD  20722 


Dr.  William  Howell 
Department  of  Psychology 
Rice  University 
Houston,  TX  77001 


Dr.  Stanley  N.  Roscoe 
New  Mexico  State  University 
Box  5095 

Las  Cruces,  NM  88003 


Dr.  Christopher  Wickens 
Department  of  Psychology 
University  of  Illinois 
Urbana,  IL  61801 


Mr.  Joseph  G.  Wohl 

Alphatech,  Inc. 

3  New  England  Executive  Park 

Burlington,  MA  01803 
% 


Mr.  Edward  M.  Connelly 
Performance  Measurement 
Associates,  Inc. 

410  Pine  Street,  S.  E. 

Suite  300 
Vienna,  VA  22180 

Professor  Michael  Athans 
Room  35-406 

Massachusetts  Institute  of 
Technology 

Cambridge,  MA  02139 

Dr.  Edward  R.  Jones 
Chief,  Human  Factors  Engineering 
McDor.nell-Douglas  Astronautics  Co. 
St.  Louis  Division 
Box  516 

St.  Louis,  MO  63166 


Dv.  Marvin  Cohen 

Decision  Science  Consortium,  Inc. 
Suite  721 

7700  Leesburg  Pike 
Falls  Church,  VA  22043 

Dr.  Robert  Wherry 
Analytics,  Inc. 

2500  Maryland  Road 
Willow  Grove,  PA  19090 

Dr.  William  R.  Uttal 
Institute  for  Social  Research 
University  of  Michigan 
Ann  Arbor,  MI  48109 

Dr.  William  B.  Rouse 
School  of  Industrial  and  Systems 
Engineering 

Georgia  Institute  of  Technology 
Atlanta,  GA  30332 
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Or.  Richard  Pew 

Bolt  Beranek  4  Newman,  Inc. 

50  Moulton  Street 
Cambridge ,  MA  02238 

Or.  Hillel  Einhorn 
Graduate  School  of  Business 
University  of  Chicago 
1101  E.  58th  Street 
Chicago,  IL  60637 

Or.  Dougias  Towne 

University  of  Southern  California 

Behavioral  Technology  Ub 

3716  S.  Hope  Street 

Los  Angeles,  CA  90007 

Or.  David  J.  Getty 

Bolt  Beranek  4  Newman,  Inc. 

50  Moulton  street 
Cambridge,  MA  02238 

Dr.  John  Payne 
Graduate  School  of  Business 
Administration 
Duke  University 
Durham,  NC  27706 

Dr.  Baruch  Fischhoff 
Decision  Research 
1201  Oak  Streec 
Eugene,  OR  97401 

Dr.  Alan  Morse 

Intelligent  Software  Systems  Inc. 
160  Old  Farm  Road 
Amherst,  MA  01002 

Dr.  J.  Miller 

Florida  Institute  of  Oceanography 
University  of  South  Florida 
St.  Petersburg,  FL  33701 


